Visualizing Video Sounds through Sound Word Animation

Sound information in video plays an important role in constructing audience experience. On the other hand, there are many circumstances where the audience cannot watch video with sounds. Subscripts are conventionally used as visual aids to provide the missing sound information. However, conventional subscripts are far less expressive for non-verbal sounds since it is designed to visualize speech sound. To address this problem, we propose a method to automatically recognize sound categories of non-verbal video-sounds and transform it to sound words for visualization. The sound word is animated in response to the change of the sound from moment to moment. Its position also changes dynamically depending on the position of the sound source object in the video. This will provide natural visual representation of non-verbal sounds with rich information about sound category and dynamics. The sound word animation is not only useful when watching video without sound, but also enjoyable for audience since it works as visual effects to enhance the visual scene.